tfEST AVAILABLE COPY 



(19) 




Europaisches Patentamt 
European Patent Office 
Office eu rope en des brevets 



1 





(11) 



EP 1 231 110 A2 



(12) 



EUROPEAN PATENT APPLICATION 



(43) Date of publication: 

14.08.2002 Bulletin 2002/33 



(51) IntCI/: B60R 1/00 



(21 ) Application number 02002896.5 

(22) Date of filing: 08.02.2002 



(84) Designated Contracting States: 


(72) 


Inventors: 


AT BE CH CY DE DK ES Fi FR GB GR IE IT LI LU 


• 


lida, Ryosuke 


MC NL PT SE TR 




Yokohama 224-0054 (JP) 


Designated Extension States: 


• 


Mizusawa, Kazufumi 


AL LT LV MK RO SI 




Kawasaki-shi, Kanagawa-ken 221-0041 (JP) 




• 


Miwa, Michio 


(30) Priority: 09.02.2001 JP 2001034093 




Urayasu-shi, Chiba-ken 279-0017 (JP) 


08.06.2001 JP 2001174415 








(74) 


Representative: 


(71 ) Applicant: MATSUSHITA ELECTRIC INDUSTRIAL 




Manitz, Finsterwald & Partner GbR 


CO., LTD. 




Postfach 31 02 20 


Kadoma-shi, Osaka 571-8501 (JP) 




80102 MUnchen (DE) 



(54) Picture synthesizing apparatus 



(57) There is disclosed a picture synthesizing appa- 
ratus for superimposing an auxiliary image upon an im- 
age converted as if the image were photographed with 
a virtual viewpoint, displaying the image in a monitor, 
and providing the image such that an approach to a sur- 
rounding solid object can easily be predicted from the 
monitor image. The picture synthesizing apparatus in- 
cludes: an image pickup unit 101 disposed in a car; a 



viewpoint change image synthesizing unit 102 for 
changing a viewpoint of the image obtained by the im- 
age pickup unit 101 and synthesizing the image; a car 
straight locus line generation unit 103 for generating a 
locus line at an arbitrary height of the car, when the car 
linearly advances; and a car locus line drawing unit 104 
for drawing the locus line generated by the car straight 
locus line generation unit 103 on the image synthesized 
by the viewpoint change image synthesizing unit 102. 
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Description 

BACKGROUND OF THE INVENTION 
FIELD OF THE INVENTION 

[0001] The present invention relates to a picture syn- 
thesizing apparatus, particularly to a picture synthesiz- 
ing apparatus preferable for use in an apparatus which 
changes a viewpoint, converts images obtained by a 
plurality of image pickup devices disposed in a car to 
images from a virtual viewpoint, synthesizes the imag- 
es, provides an image around the car to a driver, and 
helps the driver to drive the car, for example, at a parking 
time. 

DESCRIPTION OF THE RELATED ART 

[0002] As a conventional image display apparatus, an 
apparatus has been known which changes a viewpoint 
of an image actually photographed by a camera, con- 
verts the image as if the image were photographed from 
a virtual viewpoint, and displays the image in a monitor 
as disclosed in "the Principle of virtual Viewpoint Image 
Synthesis using a Road Surface Model" (the 7 th ITS In- 
ternational Conference). Moreover, an apparatus de- 
scribed in Japanese Patent No. 2610146 has been 
known which superimposes a predicted backward locus 
corresponding to a steering angle detected by a vehic- 
ular steering angle sensor upon an image of a camera 
mounted in the car. 

[0003] However, in the conventional image display 
apparatus, the predicted backward locus is superim- 
posed upon the image converted as if the image were 
photographed with the virtual viewpoint, and the image 
is displayed in the monitor. In this case, there is a prob- 
lem that a distance to a surrounding solid object is 
wrongly recognized to be longer than an actual distance 
in the image of the monitor. 

[0004] This respect will be described with reference 
to the drawing. In FIG. 30, a distance between an illus- 
trated own car 2401 and another car 2402 seems to be 
still sufficiently long in a monitor image. In actual, how- 
ever, as shown in FIG. 31, the car 2401 is considerably 
close to the other car 2402. This is because the view- 
point is changed to the virtual viewpoint, and thereby 
objects apart from the ground such as a bumper are pro- 
jected into a position farther from a camera than an ac- 
tual position in the converted image. 

SUMMARY OF THE INVENTION 

[0005] The present invention has been developed to 
solve the conventional problem, and an object of the 
present invention is to provide a picture synthesizing ap- 
paratus for superimposing an auxiliary image upon an 
image converted as if the image were photographed 
with a virtual viewpoint, and displaying the image in a 



monitor in order to provide an image such that an ap- 
proach to a surrounding solid object can easily be pre- 
dicted from the monitor image. 

[0006] According to the present invention, there is 

5 provided a picture synthesizing apparatus including: im- 
age pickup means mounted in a car; viewpoint change 
image synthesizing means for changing a viewpoint of 
an image obtained by the image pickup means and syn- 
thesizing the image; car locus line generation means for 

10 generating at least one of a locus line of an arbitrary 
height of the car and a vertical line; and car locus line 
drawing means for drawing the locus line generated by 
the car locus line generation means on the image syn- 
thesized by the viewpoint change image synthesizing 

15 means. By this constitution, the locus line of the arbitrary 
height of the car is generated, and displayed on the syn- 
thesized image obtained by changing a viewpoint, so 
that a solid locus of the car can be drawn on the synthe- 
sized image. Therefore, when the locus of the car is dis- 

20 torted similarly as distortion of a surrounding solid object 
occurring during viewpoint change and image synthe- 
sis, a three-dimensional position relation between sur- 
roundings and a predicted locus of the car can easily be 
grasped, visibility is improved, and it becomes easy to 

25 judge contact between the car and the surrounding solid 
object. 

[0007] Moreover, according to the present invention, 
there is provided a picture synthesizing apparatus in- 
cluding: image pickup means disposed in a car; view- 

30 point change image synthesizing means for changing a 
viewpoint of an image obtained by the image pickup 
means and synthesizing the image; auxiliary line gen- 
eration means for generating an auxiliary line of an ar- 
bitrary position from the car; and auxiliary line drawing 

35 means for drawing the auxiliary line generated by the 
auxiliary line generation means on the image synthe- 
sized by the viewpoint change image synthesizing 
means. By this constitution, the auxiliary line of the ar- 
bitrary position from the car is generated, and displayed 

40 on the synthesized image obtained by changing a view- 
point. The auxiliary line as a measure of a distance is 
drawn on the synthesized image, and a driver can easily 
grasp the distance from another car approaching from 
behind during driving. 

45 [0008] Furthermore, according to the present inven- 
tion, there is provided a picture synthesizing apparatus 
including: image pickup means disposed in a car; view- 
point change image synthesizing means for changing a 
viewpoint of an image obtained by the image pickup 

50 means and synthesizing the image; storage means for 
storing predetermined data beforehand; and drawing 
means for superimposing predetermined auxiliary data 
on the image synthesized by the viewpoint change im- 
age synthesizing means based on the data read from 

55 the storage means. The image pickup means is dis- 
posed so that a rear edge of the car is positioned within 
a field of view, and the viewpoint change image synthe- 
sizing means changes the viewpoint and synthesizes 
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the image including an image of the car. By the consti- 
tution, a conversion/synthesis result of the image of the 
rear edge of the car can be provided to the driver, and 
it can be easy to judge contact between the car and a 
solid object around the car 5 
[0009] According to the present invention, there is 
provided an image display apparatus comprising: dis- 
play means for displaying an image synthesized by the 
picture synthesizing apparatus; and display data con- 
version control means for converting the image, to be 10 
displayed into data suitable for the display means. 
[001 0] By this constitution, the image obtained by the 
image pickup means or the synthesized image can be 
displayed in the display means, so that an operator can 
easily grasp a position relation between a surrounding 15 
situation and the car. 

[001 1] Moreover, according to the present invention, 
there is provided an image acquirement warning appa- 
ratus comprising: detection means for detecting an ap- 
proaching state of a connection object connected to a 20 
car; and warning means for generating arbitrary warning 
information to an operator based on at least one infor- 
mation of detected information of the detection means 
and image information synthesized by the picture syn- 
thesizing apparatus. The warning information is issued 25 
in accordance with the approaching state of the arbitrar- 
ily set connection object. 

[0012] By this constitution, when the connection ob- 
ject is connected to a rear part of a car body, and the 
connection object is approaching, the warning informa- 30 
tion is issued in response to the approaching state, and 
an operator's attention can be called. Moreover, it is also 
possible to confirm a completed state at a time of com- 
pletion of connection. 

[0013] Furthermore, according to the present inven- 35 
tion, there is provided a car position recognition appa- 
ratus comprising: image detection means for detecting 
an image of an arbitrary object from an image obtained 
by rear image pickup means which is disposed behind 
a car to constitute one of image pickup means of the *o 
picture synthesizing apparatus, or an image synthe- 
sized by the picture synthesizing apparatus; recognition 
means for recognizing a position relation between the 
image detected by the image detection means and the 
image of the car; and comparison judgment means for 45 
comparison and judgment with an arbitrary set condi- 
tion. 

[0014] By the constitution, the image obtained by the 
image pickup means or the synthesized image can be 
displayed in the display means, and provided to the op- so 
erator, so that the operator can easily grasp the position 
of the object (including an object display), or the position 
relation between the object and the car from the sur- 
rounding situation, particularly from the image of the ar- 
bitrarily set object. ss 



BRIEF DESCRIPTION OF THE DRAWINGS 
[0015] 

FIG. 1 is a block diagram showing a constitution of 
a picture synthesizing apparatus according to a first 
embodiment of the present invention; 
FIG. 2 is an explanatory view of a synthesized im- 
age and focus line obtained by the picture synthe- 
sizing apparatus of FIG. 1 and viewed vertically 
downwards; 

FIG. 3 is a block diagram showing a constitution of 
locus line calculation means 103B having an arbi- 
trary height in FIG. 1; 

FIG. 4 is an explanatory view of a flow of processing 
of the locus line calculation means 103B having the 
arbitrary height in FIG. 1; 

FIG. 5 is an explanatory view of details of the locus 
line of FIG. 2. 

FIG. 6 is an explanatory view of a rear panorama 
synthesized image and the locus tine obtained by 
the picture synthesizing apparatus of FIG. 1; 
FIG. 7 is a block diagram showing the constitution 
of the picture synthesizing apparatus according to 
a second embodiment of the present invention; 
FIG. 6 is a block diagram showing the constitution 
of calculation means 502B of the locus line having 
the arbitrary height in FIG. 7; 
FIG. 9 is an explanatory view of the synthesized im- 
. age and locus line obtained by the picture synthe- 
sizing apparatus of FIG. 7 and viewed backwards; 
FIG. 10 is an explanatory view of the synthesized 
image and locus line obtained by the picture syn- 
thesizing apparatus of FIG. 7 and viewed vertically 
downwards; 

FIG. 11 is an explanatory view of a solid display of 
the synthesized image, locus line, and car moving 
with an elapse of time, obtained by the picture syn- 
thesizing apparatus of FIG. 7 and viewed vertically 
downwards; 

FIG. 12 is a block diagram showing the constitution 
of the picture synthesizing apparatus according to 
a third embodiment of the present invention; 
FIG. 13 is an explanatory view of an emphasized 
display of the rear panorama synthesized image, lo- 
cus line, and collision place obtained by the picture 
synthesizing apparatus of FIG. 12; 
FIG. 14 is a block diagram showing the constitution 
of the picture synthesizing apparatus according to 
a fourth embodiment of the present invention; 
FIG. 15 is an explanatory view of a multi-screen dis- 
play image and locus line obtained by the picture 
synthesizing apparatus of FIG. 14; 
FIG. 16 is a block diagram showing the constitution 
of the picture synthesizing apparatus according to 
a fifth embodiment of the present invention; . 
FIG. 17 is an explanatory view of a multi-screen dis- 
play image and auxiliary line obtained by the picture 
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synthesizing apparatus of FIG. 16; 
FIG. 18 is a block diagram showing the constitution 
of an image synthesis/display apparatus according 
to a sixth embodiment of the present invention; 
FIG. 19 is a layout of an image pickup unit of the 
image synthesis/display apparatus according to the 
sixth embodiment of the present invention; 
FIG. 20 is a screen constitution diagram of the im- 
age synthesis/display apparatus according to the 
sixth embodiment of the present invention; 
FIG. 21 is a screen constitution diagram of the im- 
age synthesis/display apparatus according to a 
seventh embodiment of the present invention; 
FIG. 22 is a screen constitution diagram of the im- 
age synthesis/display apparatus according to an 
eighth embodiment of the present invention; 
FIG. 23 is a screen constitution diagram of the im- 
age synthesis/display apparatus according to a 
ninth embodiment of the present invention; 
FIG. 24 is a block diagram showing the constitution 
of the image synthesis/display apparatus according 
to a tenth embodiment of the present invention; 
FIG. 25 is a screen constitution diagram of the im- 
age synthesis/display apparatus according to an 
eteventh embodiment of the present invention; 
FIG. 26 is a screen constitution diagram of the im- 
age synthesis/display apparatus according to a 
twelfth embodiment of the present invention; 
FIG. 27 is a block diagram showing the constitution 
of the image synthesis/display apparatus according 
to a thirteenth embodiment of the present invention; 
Fl G . 28 is a block diagram of an i mage acquirement/ 
warning apparatus according to a fourteenth em- 
bodiment of the present invention; 
FIG. 29 is a block diagram of a car position recog- 
nition apparatus according to a fifteenth embodi- 
ment of the present invention; 
FIG. 30 is a screen constitution diagram of a con- 
ventional rear view field display apparatus; and 
FIG. 31 is a position relation diagram of a car show- 
ing a problem of the conventional rear view field dis- 
play apparatus. 

DETAILED DESCRIPTION OF THE PREFERRED 
EMBODIMENTS 

[0016] Embodiments of the present invention will be 
described hereinafter in detail with reference to the 
drawings. 

(First Embodiment) 

[0017] FIG. 1 is a block diagram showing a constitu- 
tion of a picture synthesizing apparatus according to a 
first embodiment. The picture synthesizing apparatus is 
constituted of image pickup means 101 disposed in a 
car, viewpoint change image synthesizing means 102, 
car straight locus line generation means 103, and car 
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locus line drawing means 104. 

[0018] The image pickup means 101 is constituted of 
one or more cameras disposed in a rear or side part of 
the car, and turns to the rear of the car. The viewpoint 

* 

5 change image synthesizing means 102 inputs one or 
more images obtained by the image pickup means 101 , 
and outputs a synthesized image viewed from a virtual 
viewpoint. The car straight locus line generation means 
103 inputs a size and shape of the car, camera param- 

10 eter for changing the viewpoint, and image pickup situ- 
ation of the image pickup means 101 , and outputs a lo- 
cus line in a case in which the car runs straight back- 
wards. The car straight locus line generation means 103 
includes locus line calculation means 103A on a road 

15 surface and locus line calculation means 103B having 
an arbitrary height. The means can output the locus line 
of an arbitrary position of the car, and has a function of 
outputting, for example, a locus on the road surface, lo- 
cus of a bumper end of the car, and the like at the same 

20 time. Car locus line drawing means 104 superimposes 
and draws the locus line of the car outputted by the car 
straight locus line generation means 103 onto the syn- 
thesized image obtained by the viewpoint change image 
synthesizing means 102. Additionally, an output of the 

25 car locus line drawing means 104 is supplied and dis- 
played into a display device (not shown). 
[0019] FIG. 2 shows that the image obtained by the 
image pickup means 101 is synthesized to obtain an im- 
age viewed downwards vertically to the road surface 

30 with the car laid thereon from a rear of the car by the 
viewpoint change image synthesizing means 102, and 
a locus line 201 on the road surface of a car 204 and 
locus line 202 of the car at a height of 50 cm from the 
road surface obtained by the car locus line drawing 

35 means 104 are drawn on the synthesized image by the 
car straight locus line generation means 103. For ease 
of understanding that a solid object is projected on the 
synthesized image, the locus line on the road surface 
201 and the locus line having the height 202 are con- 

40 nected to each other via a straight line vertical to the 
road surface 203. Here, another first car 205 exists be- 
hind the car 204, and another second car 206 exists be- 
side the car. 

[0020] FIG. 3 is a block diagram showing a constitu- 
45 tion of the locus line calculation means 103B having the 
arbitrary height in FIG. 1 . The arbitrary-height locus line 
calculation means 103B is constituted of three-dimen- 
sional locus line generation means 103B-1, three-di- 
mensional locus line road surface projection means 
50 103B-2, and synthesized image projection means 
103B-3. The three-dimensional locus line generation 
means 1 03B-1 has a function of generating a three-di- 
mensional locus line of the car at the arbitrary height in 
a real space. The three-dimensional locus line road sur- 
55 face projection means 103B-2 has a function of project- 
ing the three-dimensional locus line generated by the 
three-dimensional locus line generation means 103B-1 
onto the road surface from the image pickup means at- 
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tached to the car. The synthesized image projection 
means 103B-3 has a function of projecting the locus line 
projected on the road surface by the three-dimensional 
locus line road surface projection means 103B-2 onto 
the synthesized image. 5 
[0021] FIG. 4 is an explanatory view of a flow of 
processing of the arbitrary-height locus line calculation 
means 103B. First, the three-dimensional locus Ijne 
generation means 103B-1 generates a line segment 
105 as a locus line of an arbitrary portion of the car in a 10 
three-dimensional space. Subsequently, the line seg- 
ment 105 is projected onto a line segment 106 on the 
road surface from the image pickup means attached to 
the car by the three-dimensional locus line road surface 
projection means 103B-2. Next, the line segment 106 is ts 
projected onto a line segment 107 on the synthesized 
image by the synthesized image projection means 
103B-3. The line segment 107 obtained in this manner 
is a locus line on the synthesized image. 
[0022] FIG. 5 is a diagram showing the locus line of 20 
FIG. 2 in detail. As shown in FIG. 5, it is assumed that 
respective points on the locus of the car on the road sur- 
face are a to h, and respective points on the locus at a 
height of 50 cm from the road surface are A to H. 
[0023] A line ab is a straight line indicating a position 25 
of the rear end of the car 204 on the road surface; line 
cd is a straight line indicating a position of 1 m behind 
the car 204 on the road surface; line ef is a straight line 
indicating a position of 2 m behind the car 204 on the 
road surface; and line gh is a straight line indicating a 30 
position of 3 m behind the car 204 on the road surface. 
Moreover, a line AB is a straight line indicating a height 
of 50 cm of the rear end of the car 204; line CD is a 
straight line indicating a height of 50 cm at 1 m behind 
the car 204; line EF is a straight line indicating a height 35 
of 50 cm at 2 m behind the car 204; and line GH is a 
straight line indicating a height of 50 cm at 3 m behind 
the car 204. Therefore, it can be supposed that lines Aa, 
Bb, Cc, Dd, Ee, Ff, Gg, Hh indicate images of rods each . 
having a length of 50 cm and standing vertically to the *o 
road surface in respective positions on the road surface 
on the synthesized image. 

[0024] With the locus lines determined as described 
above, for example, for only the road surface displayed 
in a region inside a square ABCD, it can be said that an 
object having a height of 50 cm or less is not present at 
least in a width of the carand at a distance of 1 m behind. 
Conversely, for a solid object other than the road surface 
displayed in the square ABCD, it can be said that there 
is a possibility of presence of the object having the so 
height of 50 cm or less in the width of the car and at the 
distance of 1 m behind. 

[0025] Here, in the apparatus of FIG. 1 , it is also pos- 
sible to freely change a color of the locus line to be 
drawn. For example, it is possible to draw a line segment 55 
indicating a range of 1 m from the car in red, draw a line 
segment indicating a range of 1 m to 2 m in yellow, and 
draw a line segment indicating a range of 2 m to 3 m in 



blue. Moreover, loci aceg and bdfh on the road surface 
are drawn in black or bold lines, and it is then possible 
to display the loci of car tires so that the loci can easily 
be seen by intuition. 

[0026] Moreover, in the apparatus of FIG. 1 , it is also 
possible to draw the locus of the height of a car bumper. 
When the locus of the height of the car bumper is drawn, 
it is easy to judge a contact of the car bumper with a 
surrounding object on the synthesized image. 
[0027] Furthermore, the locus line is drawn 3 m at 
maximum behind the car in FIG. 2, but the locus line can 
be drawn for an arbitrary distance. 
[0028] FIG. 6 shows that a rear panorama image is 
synthesized from the image obtained by the image pick- 
up means 101 by the viewpoint change image synthe- 
sizing means 1 02, and a locus 401 of the car on the road 
surface, locus 402 of a bumper end of the car, and locus 
403 of the height of the car are drawn on the synthesized 
image. When the locus of the height of the car is drawn, 
it is easy to grasp a sense of distance from a surrounding 
car. 

[0029] As described above, according to the first em- 
bodiment of the present invention, since it is possible to 
draw the solid locus of the car on the viewpoint changed 
synthesized image, a driver can easily judge the contact 
of the car with the surrounding solid object. 
[0030] Moreover, when the locus line of the car on the 
road surface and the locus line at the arbitrary height 
are interpolated with a straight or curved line, it is also 
possible to display the line vertical to the road surface 
or a sectional view of the car on the locus line on the 
synthesized image. 

[0031] For example, a scene is assumed in which the 
car is connected to a load earner using a load carrier 
hitch attached to the car. In an initial state in which the 
load carrier is detached from the car, that is, when the 
car is brought close to the load carrier, a course of action 
for bringing the car close to the load carrier as an object 
is easily established by a displayed presumed locus of 
the car. Moreover, just before the load carrier is connect- 
ed, the image is converted as if the image were viewed 
right from above by changing the viewpoint (plane pro- 
jection). It is then possible to exactly grasp a position 
relation between the hitches. By the aforementioned op- 
eration and use, the load carrier can easily be connected 
to the car. 

[0032] Moreover, during parking into a limited parking 
space, or during grasping of the position relation of fol- 
lowing cars in a plurality of traffic lanes, the image easily 
understandable for the driver can similarly advanta- 
geously be provided. 

[0033] Furthermore, in the example described above, 
when the height of a most critical portion, such as the 
height of the hitch or the bumper, is known beforehand, 
the locus line at the height may be drawn in response 
to a driver's instruction. 

[0034] Additionally, when the locus line of the bumper 
end is drawn, it is possible to judge the contact of the 
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bumper end. Moreover, when the locus line of the car 
height is drawn onto the synthesized image, it is possi- 
ble to draw the locus line easily understandable, by in- 
tuition. 

[0035] Furthermore, when the locus line is drawn with 
various color or thickness in accordance with the dis- 
tance from the car, a driver's attention can effectively be 
concentrated in the vicinity of the car. 
[0036] The picture synthesizing apparatus according 
to the first embodiment of the present invention can also 
be used as a superior driving support apparatus having 
safety for clearly indicating a blind spot in the rear to the 
driver and convenience during parking or connecting. 

(Second Embodiment) 

[0037] FIG. 7 is a block diagram showing the consti- 
tution of the picture synthesizing apparatus according 
to a second embodiment. In FIG. 2, the same constitut- 
ing elements as those of FIG. 1 or the corresponding 
constituting elements are denoted with the same refer- 
ence numerals as those of FIG. 1. 
[0038] The picture synthesizing apparatus is consti- 
tuted by disposing car steering angle corresponding lo- 
cus line generation means 502 instead of the car straight 
locus line generation means 103, adding steering angle 
information output means 501, and inputting an output 
of the means to the car steering angle corresponding 
locus line generation means 502 in the picture synthe- 
sizing apparatus of FIG. 1. Other constituting elements 
have the same constitution as that of the apparatus of 
FIG. 1. 

[0039] The steering angle information output means 
501 is a device for outputting a steering wheel angle of 
the car. The car steering angle corresponding locus line 
generation means 502 is a device for inputting the steer- 
ing wheel angle and outputting the locus line corre- 
sponding to the steering wheel angle. The car steering 
angle corresponding locus line generation means 502 
includes calculation means 502A of the locus line on the 
road surface and calculation means 502B of the locus 
line at the arbitrary height. The means has a function of 
inputting the steering wheel angle, size and shape of the 
car, camera parameter for changing the viewpoint, and 
image pickup situation of the image pickup means 101, 
and outputting the locus line corresponding to the steer- 
ing angle in the arbitrary position of the car, when the 
car runs backwards. 

[0040] FIG. 8 is a block diagram showing the consti- 
tution of the calculation means 502B of the locus line at 
the arbitrary height in FIG. 7. The calculation means 
502B of the locus line at the arbitrary height is constitut- 
ed of three-dimensional shape storage means 502B-1, 
three-dimensional locus region generation means 
502B-2, road surface projection means 502B-3, and 
synthesized image projection means 502B-4. The 
three-dimensional shape storage means 502B-1 can 
store the shape of the car, and the shape of a rectangu- 



lar parallelepiped or a part of the car inscribed by the 
car such as the bumper and wheel. The three-dimen- 
sional locus region generation means 502B-2 has a 
function of generating a region in the real space through 

5 which the shape of the car stored in the three-dimen- 
sional shape storage means 502B-1 passes during 
movement of the car The road surface projection 
means 502B-3 has a function of projecting the three- 
dimensional region generated by the three-dimensional 

10 locus region generation means 502B-2 onto the road 
surface. The synthesized image projection means 
502B-4 has a function of projecting the three-dimension- 
al region generated by the road surface projection 
means 502B-3 onto the synthesized image. Additional- 

*5 |y, the shape stored by the three-dimensional shape 
storage means 502B-1 may be a point of a certain por- 
tion of the car. Therefore, the region generated by the 
three-dimensional locus region generation means 
502B-2 can also be a three-dimensional line segment. 

20 [0041] FIG. 9 shows that the locus line is drawn onto 
the synthesized image obtained by changing the view- 
point and synthesizing the image in accordance with 
steering angle information. Here, the three-dimensional 
shape storage means 502B-1 in the car steering angle 

25 corresponding locus line generation means 502 stores 
the three-dimensional position of a point having a height 
of 50 cm from the road surface in a corner behind the 
car. As shown in FIG. 9, a locus line 601 corresponding 
to the steering angle on the road surface, and a locus 

30 line 602 corresponding to the steering angle at the 
height of 50 cm are displayed. By such display, it is easy 
to judge the steering wheel angle of the car at which the 
car contacts the surrounding solid object. The contact 
with the surrounding solid object can easily be avoided 

35 during parking. 

[0042] FIG. 10 shows that the apparatus of FIG. 7 is 
applied to a rear panorama image. Here, two other cars, 
a steering angle corresponding locus line 701 on the 
road surface, a steering angle corresponding locus line 

40 702 at the height of the bumper, and a steering angle 
corresponding locus line 703 of the car height are dis- 
played behind the car. 

[0043] FIG. 11 shows that the tires and bumper of the 
car during backward movement of the car are projected 

45 on the road surface and drawn on the synthesized im- 
age. Here, the three-dimensional shape storage means 
502B-1 stores the shapes of the bumper and wheels of 
the car. The car steering angle corresponding locus line 
generation means 502 projects a projected image 801 

50 of the wheel of the car 204, and a projected image 802 
of the bumper end. The projected image moves back- 
wards from the rear end of the car 204 with an elapse 
of time in a certain period. Thereby, it is possible to easily 
grasp the position relation with the surrounding object 

55 by intuition, when the car 204 moves. Particularly, when 
the steering wheel is turned, the locus of the tire and the 
locus of the bumper differ because of a difference of a 
rotation radius. Therefore, it is effective to draw a solid 
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model of the car onto the synthesized image as shown 
in FIG. 8. 

[0044] As described above, according to the second 
embodiment of the present invention, the solid locus of 
the car is drawn in accordance with the steering angle. 
Therefore, the turning angle of the car steering wheel at 
which the car contact the surrounding solid object can 
easily be determined, and the contact with the surround- 
ing solid object can easily be avoided during the parking. 
[0045] Moreover, the section or the solid model of the 
car moved along the locus line with the elapse of time 
so that the car moves apart from the rear end of the car 
is drawn, and the contact with the solid object can easily 
be judged. 

(Third Embodiment) 

[0046] FIG. 12 is a block diagram showing the consti- 
tution of the picture synthesizing apparatus according 
to a third embodiment. In FIG. 12, the same constituting 
elements as those of FIG. 7 or the corresponding con- 
stituting elements are denoted with the same reference 
numerals as those of FIG. 7. 

[0047] The picture synthesizing apparatus is consti- 
tuted by adding obstacle collision prediction means 901 
and obstacle collision emphasis display means 902 to 
the apparatus of FIG. 7, and other constituting elements 
have a constitution similar to the constitution of the ap- 
paratus of FIG. 7. 

[0048] The obstacle collision prediction means 901 is 
a device for detecting obstacles such as a car present 
around the car. Moreover, the car locus line drawing 
means 1 04 has a function of not drawing the subsequent 
locus line, when the position of the obstacle detected by 
the obstacle collision prediction means 901 is judged to 
collide against the locus line of the car. The obstacle 
collision emphasis display means 902 has a function of 
emphasizing/displaying a collision place on the synthe- 
sized image, when the locus line of the car is judged to 
collide against the obstacle. 

[0049] In the constitution, in the apparatus of FIG. 12, 
with the obstacle present on the predicted locus line, un- 
naturalness with which the locus is drawn on the obsta- 
cle is eliminated. Moreover, it is possible to display the 
predicted locus line which can more easily be under- 
stood. 

[0050] FIG. 13 shows a panorama image and locus 
line obtained by the apparatus of FIG. 12. In FIG. 13, a 
dotted-line portion 1001 of the locus line present before 
the obstacle is not drawn on the image. Moreover, since 
the collision place is emphasized/displayed as shown 
by 1002, it is possible to concentrate the driver's atten- 
tion onto the collision place. 

[0051] As described above, according to the third em- 
bodiment of the present invention, the locus line is not 
drawn on the obstacle around the car, and it is therefore 
possible to prevent an unnatural phenomenon of draw- 
ing the locus on a route along which the car cannot orig- 
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inaily advance. Therefore, it is possible to draw the locus 
line with which the surrounding situation can easily be 
seen. 

[0052] Moreover, when the collision of the car against 
5 the obstacle is predicted, the collision place is empha- 
sized and displayed, and the driver's attention can be 
called. 

(Fourth Embodiment) 

10 

[0053] FIG. 14 is a block diagram showing the consti- 
tution of the picture synthesizing apparatus according . 
to a fourth embodiment. In FIG. 14, the same constitut- 
ing elements as those of FIG. 7 or the corresponding 

15 constituting elements are denoted with the same refer- 
ence numerals as those of FIG. 7. 
[0054] The picture synthesizing apparatus is consti- 
tuted by adding multi-screen generation means 1101 to 
the picture synthesizing apparatus of FIG. 7, and other 

20 constituting elements have the constitution similar to 
that of the apparatus of FIG. 7. Additionally, the image 
pickup means 101 is disposed in the rear part and op- 
posite side parts of the car to obtain an image for gen- 
erating multiple screens. 

25 [0055] FIG. 15 is a diagram showing a synthesized 
image obtained based on the images from the cameras 
disposed in the rear part and opposite side portions of 
the car by the picture synthesizing apparatus of FIG. 14, 
and the car locus lines. About 2/3 upper part of a display 

30 region is used to display the images of the camera dis- 
posed in the rear part of the car, and about 1/3 lower 
part of the display region is used to display the images 
from the cameras disposed in the opposite side portions 
of the car. Moreover, a locus line 1201 behind the car is 

35 drawn in each screen. Furthermore, a locus 1202 in a 
front comer of the car can also be drawn on the image 
of the camera which turns to the side of the car. Here, 
the locus lines or car frames indicating the same position 
in different screens are displayed in the same color, so 

40 that it is preferably easy to handle the locus of the same 
position of the car even in the different screens. 
[0056] As described above, according to the fourth 
embodiment of the present invention, since the loci be- 
hind the car are drawn on the images obtained from the 

45 cameras in two or more different positions, the three- 
dimensional position of the locus line can easily be seen, 
and it is easy to judge the contact with the surrounding 
obstacle. 

[0057] Moreover, since the locus lines or the car 
so frames indicating the same position in the different 
screens are displayed in the same color, it is easy to 
handle the loci of the same position of the car even in 
the different screens. 

[0058] Additionally, the apparatus of FIG. 14 can have 
55 a function of moving a frame 1203 of the car with time. 
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(Fifth Embodiment) 

[0059] FIG. 16 is a block diagram showing the consti- 
tution of the picture synthesizing apparatus according 
to a fifth embodiment. In FIG. 16, the same constituting 
elements as those of FIG. 1 or the corresponding con- 
stituting elements are denoted with the same reference 
numerals as those of FIG. 1. 

[0060] The picture synthesizing apparatus is consti- 
tuted by adding the multi-screen generation means 
1101 to the picture synthesizing apparatus of FIG. 1 , fur- 
ther replacing the locus line generation means 103 with 
auxiliary line generation means 1301 , and replacing the 
car locus line drawing means 104 with auxiliary line 
drawing means 1302. The other constituting elements 
have the same constitution as that of the apparatus of 
FIG. 1. 

[0061] The auxiliary line generation means 1301 is a 
device for generating lines indicating positions 10m and 
20 m behind the car on the synthesized image. Moreo- 
ver, the auxiliary line drawing means 1302 superimpos- 
es and draws auxiliary lines outputted from the auxiliary 
line generation means 1301 onto the synthesized image 
obtained by the multi-screen generation means 1101. 
[0062] FIG. 17 shows the synthesized image obtained 
by the picture synthesizing apparatus of FIG. 16. Here, 
an auxiliary line 1401 at 10 m behind the car, and an 
auxiliary line 1402 at 20 m behind the car are displayed. 
It is easy to grasp the distance from the car approaching 
from behind by the auxiliary lines 1401 and 1402. More- 
over, it is also possible to draw an auxiliary line 1403 
obtained by projecting the line indicating the rear end of 
the car vertically onto the road surface. It is easily judged 
by the auxiliary line 1403 whether another car trying to 
get ahead of the car is running behind the rear end of 
the car or beside the car. Moreover, another car running 
beside the car can smoothly be checked during chang- 
ing of the traffic lane. Furthermore, an auxiliary line 1404 
indicating a width of a general car can be displayed on 
the auxiliary lines 1401, 1402, 1403. Thereby, the car 
approaching from behind runs close to a certain scale 
line, and the width of the car becomes substantially the 
same as the auxiliary line 1403. It is then seen that the 
car is running in the vicinity of the auxiliary line. 
[0063] As described above, according to the fifth em- 
bodiment of the present invention, when the auxiliary 
line is drawn as a measure of the distance on the syn- 
thesized image, it is easy for the driver to grasp the. dis- 
tance from the car approaching from behind during run- 
ning. 

[0064] Moreover, the auxiliary line indicating the po- 
sition of the rear end of the car is generated on the image 
obtained by photographing from the side of the car, or 
the synthesized image of the image. Therefore, it is easy 
to see that the front end of another car running beside 
the car is present behind or before the rear end of the 
car, and this is effective during the changing of the traffic 
lane. 



[0065] Furthermore, since the auxiliary line indicating 
the general car width is drawn on these auxiliary lines, 
it is possible to more easily grasp the distance from the 
surrounding car. 

5 

(Sixth Embodiment) 

[0066] FIG. 18 is a block diagram showing the consti- 
tution of an image synthesis/display apparatus accord- 
10 jng to a sixth embodiment. The image synthesis/display 
apparatus is constituted of a first image pickup unit 1501 
and second image pickup unit 1502 disposed in the car, 
an image conversion/synthesis unit 1 503, a drawing unit 

1504, a memory 1505, and a display 1506. Here, the 
15 first image* pickup unit 1501 and second image pickup 

unit 1502 correspond to the image pickup means 101 of 
FIG. 1, and the image conversion/synthesis unit 1503 
corresponds to the viewpoint change image synthesiz- 
ing means 102. 
20 [0067] The first and second image pickup units 1501 
and 1502 photograph the rear of the car. The images 
obtained by the first and second image pickup units 

1501 and 1502 are converted to an image photographed 
from a virtual viewpoint by the image conversion/syn- 

25 thesis unit 1503. The drawing unit 1504 draws illustra- 
tion, auxiliary line, and the like to be superimposed upon 
the output image based on data stored in the memory 

1505. The image drawn by the drawing unit 1504 is dis- 
played in the display 1506. Additionally, FIG. 18 shows 

30 two image pickup units, but any number of units may be 
disposed as long as the units can be handled by the im- 
age conversion/synthesis unit 1503. As shown in FIG. 
19, the first and second image pickup units 1501 and 

1 502 are disposed in the rear part of the car 204 so that 
35 the bumper of the car 204 and a part of the car body 

behind are positioned within a field of view of the units. 
[0068] A screen constitution of the image synthesis/ 
display apparatus constituted as described above will 
be described with reference to FIG. 20. FIG. 20 shows 

40 a display example of conversion to an image displayed 
as if the image pickup unit were disposed in a position 
apart from the car rear end by about 2 m. 
[0069] Here, the car 204, another first car 205 behind, 
another second car 206 positioned beside and behind, 

45 and road surface 1701 are displayed. A meshed portion 
1 702 is a road surface outside the view field of the image 
pickup unit. 

[0070] Moreover, a rear number plate 204A, bumper 
204B, and stop lamp 204C of the car 204 are displayed. 

50 Furthermore, an auxiliary line 1703 is superimposed up- 
on the rear end of the bumper and a rear edge position 
of a car body to emphasize the rear edge of the car. In 
this case, the auxiliary is drawn outside an actual rear 
edge, or a bold auxiliary line is drawn, so that a danger 

55 of contact can be reduced. 

[0071] Moreover, a mirror confirmation line 1704 is 
drawn at a constant distance behind the rear end of the 
bumper 204B and horizontally with the bumper 204B. 
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The mirror confirmation line 1704 is an auxiliary line for 
separating the displayed image into two regions: a dan- 
gerous region within the same region as that of the car, 
and a safe region in a region different from that of the 
car. The line has a function of urging the driver to check 
collision with the surrounding object by a mirror. That is, 
when the object displayed around the car contacts the 
mirror confirmation line 1704 during driving, the driver 
can check possibility of the collision with the surrounding 
object at an optimum timing. Additionally, the mirror con- 
firmation line 1704 may be a curve drawn outside the 
rear edge of the body and the rear end of the bumper 
displayed in the image by a constant distance, or a curve 
enclosing the rear edge of the car. 
[0072] As described above, according to the sixth em- 
bodiment of the present invention, the image pickup unit 
is disposed so that the bumper of the car and a part of 
the car behind are positioned within the view field, and 
the image including the image of the car is subjected to 
viewpoint change, and synthesized. It is then easy to 
judge the contact between the car and the solid object 
around the car. 

[0073] Moreover, the auxiliary line is superimposed 
upon the positions of the rear end of the bumper and 
rear edge of the car, and the image for emphasizing/ 
displaying the positions is provided. Therefore, the rear 
edge of the car can easily be distinguished, and the con- 
tact can more easily be judged. 

[0074] Furthermore, the image is provided in which 
the mirror confirmation line is displayed behind the rear 
end of the bumper of the car by the constant distance 
and horizontally with the bumper. Therefore, the driver 
can be urged to check the safety around the car by the 
mirror at the optimum timing. 

(Seventh Embodiment) 

[0075] The constitution of the image synthesis/display 
apparatus of a seventh embodiment is the same as that 
of FIG. 18. FIG. 21 is a screen constitution diagram ac- 
cording to the seventh embodiment. 
[0076] in FIG. 21, a three-dimensional illustration 
1801 of the car is displayed. The illustration 1801 is pre- 
pared as the converted image viewed from the virtual 
viewpoint in a position apart from the car rear end by 
about 2 m, and is superimposed upon a region corre- 
sponding to that of the car of the displayed image. 
[0077] Moreover, an auxiliary line 1802 indicates a 
bumper region. In the region, an image obtained by con- 
verting the image actually obtained by the image pickup 
unit may be displayed. Thereby, a sense of security that 
the actual image is viewed can be given to the driver. In 
this case, the illustration of the bumper region is drawn 
larger than the actual car position, so that the danger of 
contact can be reduced. 

[0078] Furthermore, an auxiliary line 1803 is a locus 
of passage on the road surface indicating a position ob- 
tained by projecting the position passed by the body end 



onto the road surface, when the car linearly runs back- 
wards. Moreover, an auxiliary line 1804 is a bumper pas- 
sage locus indicating the position passed by the bumper 
end, when the car linearly runs backwards. Further- 

5 more, auxiliary lines 1805A to 1805C are solid auxiliary 
lines which connect points of the passage locus on the 
road surface at the constant distance from the car to 
points on the bumper end passage locus. Additionally, 
it is possible to display not only the passage locus of the 

10 bumper end of the car, but also the passage locus of the 
body end of the car corresponding to the height of the 
bumper of another appropriate car which height could 
be different from that of the car. 

[0079] As described above, according to the seventh 
15 embodiment of the present invention, the three-dimen- 
sional illustration prepared as the car image obtained by 
the actually disposed image pickup unit and converted/ 
synthesized is superimposed upon the displayed image, 
and the car is displayed in a solid manner in the image. 
20 Therefore, it is easy to grasp the position relation be- 
tween the car and the solid object around the car in a 
three-dimensional manner. As a result, contact judg- 
ment is facilitated. 

[0080] Moreover, the image obtained by transforming/ 
25 synthesizing the actually taken image is displayed in the 
region of the illustration corresponding to the bumper, 
and therefore the sense of security that the actual image 
is viewed can be given to the driver. 
[0081] Furthermore, the displayed image is provided 
30 which includes the road surface passage locus indicat- 
ing the position obtained by projecting the position 
passed by the body end onto the road surface, the 
bumper end passage locus indicating the position 
passed by the bumper end during the backward driving 
35 of the car, and the solid auxiliary line for connecting 
these loci to generate a solid sense. Therefore, it be- 
comes easy to grasp the position relation between two 
passage loci in a solid manner. 

40 (Eighth Embodiment) 

[0082] The constitution of the image synthesis/display 
apparatus of an eighth embodiment is the same as that 
of FIG. 18. FIG. 22 is a screen constitution diagram ac- 

45 cording to the eighth embodiment. 

[0083] In FIG. 22, an illustration 1901 is a three-di- 
mensional illustration representing a skeleton of the car 
and also representing the tire so that a position of the 
car tire in contact with the road surface can be checked. 

50 The illustration 1901 is prepared as the image viewed 
from the virtual viewpoint disposed in a position apart 
from the car rear end by about 2 m, and is superimposed 
upon the region corresponding to the car in the dis- 
played image. Additionally, the car may be represented 

55 in a wire frame. 

[0084] Moreover, an auxiliary line 1902 indicates the 
bumper region. The image obtained by converting the 
image actually obtained by the image pickup unit may 
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be displayed in the region. Furthermore, an auxiliary line 
1903 is a road surface passage locus drawn from the 
tire in the illustration 1901 . Thereby, the meaning of the 
road surface passage locus can easily be grasped by 
intuition. 5 
[0085] As described above, according, to the eighth 
embodiment of the present invention, there is provided 
the image in which the illustration of the car is represent- 
ed in the skeleton or the wire frame, and the tire position 
is explicitly displayed. Thereby, the solid sense of the 10 
car can be emphasized, and the tire position on the road 
surface can be understood by intuition. With the con- 
verted image viewed from the virtual viewpoint in the po- 
sition apart from the car rear end by about 2 m, the po- 
sition relation between the car and the solid object 15 
around the car can easily be grasped in the three-di- 
mensional manner, and the contact judgment is facilitat- 
ed. 

(Ninth Embodiment) 20 

[0086] The constitution of the image synthesis/display 
apparatus of a ninth embodiment is the same as that of 
FIG. 18. FIG. 23 is a screen constitution diagram ac- 
cording to the ninth embodiment. 25 
[0087] In FIG. 23, a first wall surface 2001 is an illus- 
tration of a wall surface which is disposed vertically to 
the road surface in the rear end position of the car and 
which has the same width as that of the car. The wall 
surface 2001 is prepared as the converted image 30 
viewed from the virtual viewpoint in the position apart 
from the car rear end by about 2 m, and superimposed 
upon the region corresponding to the rear end position 
of the car in the displayed image. Moreover, a second 
wall surface 2002 is an illustration of the wall surface 35 
which is laid inside the car and which has the same width 
as the car width. The second wall surface 2002 is pre- 
pared similarly as the first wall surface 2001 , and super- 
imposed upon an appropriate position in the displayed 
image. Here, the second wall surface 2002 is drawn so *o 
that the contact portion with the road surface is visible. 
[0088] Additionally, it is also possible to. draw an illus- 
tration of the car rear part in the first wall surface 2001, 
or to draw an illustration of the tire in the contact portion 
of the second wall surface 2002 with the road surface. 45 
[0089] As described above, according to the ninth em- 
bodiment of the present invention, there is provided the 
image in which the illustrations of two wall surfaces dis- 
posed in the rear end position of the car and inside the 
car from the rear end position and vertically to the road 50 
surface, and having the same width as that of the car 
* are superimposed upon the displayed image, and the 
car is metaphorically represented by the solid object. 
Thereby, the car can be grasped in the solid manner. 
Therefore, the position relation between the car and the 55 
solid object around the car can easily be grasped in the 
three-dimensional manner, and the contact judgment is 
facilitated. 



(Tenth Embodiment) 

[0090] FIG. 24 is a block diagram showing the consti- 
tution of the image synthesis/display apparatus accord- 
ing to a tenth embodiment. In FIG. 24, the same consti- 
tuting elements as those of FIG. 18 or the corresponding 
constituting elements are denoted with the same refer- 
ence numerals as those of FIG. 18. 
[0091] In FIG. 24, a locus calculation unit 2101 calcu- 
lates the locus of passage of the car moving backwards 
with the current steering angle based on a steering 
wheel angle signal of the car inputted from the outside. 
The drawing unit 1504 draws the illustration, auxiliary 
line, and the like to be superimposed upon the output 
image based on the data stored in the memory 1505, 
and the predicted passage locus of the car inputted from 
the locus calculation unit 2101. 

[0092] In the screen constitution of the tenth embod- 
iment, the auxiliary lines 1803, 1804, 1805A to 1805C 
in the screen of FIG. 21 are curved in accordance with 
the steering angle like the auxiliary lines 601 , 602 in FIG. 

9. 

[0093] As described above, according to the tenth 
embodiment of the present invention, there is provided 
the displayed image which includes the road surface 
passage locus indicating the position obtained by pro- 
jecting the position passed by the body end of the car 
onto the road surface in accordance with the steering 
angle of the car, the bumper end passage locus indicat- 
ing the position passed by the bumper end of the car, 
and the solid auxiliary line for connecting the loci to pro- 
duce the solid sense. Thereby, in the backward move- 
ment, the actual passage predicted locus can be dis- 
played in the solid manner in accordance with the steer- 
ing angle of the car so that the locus can easily be 
grasped. The driver can refer to the displayed screen to 
operate the steering wheel in an optimum manner. 
[0094] Additionally, these auxiliary lines may be dis- 
played together with the actual image or the illustration 
of the car as shown in the sixth to ninth embodiments 
(FIGS. 20 to 23). The road surface passage locus may 
be displayed like a tire trace. Alternatively, the bumper 
end passage locus may be connected to the bumper 
end of the actual image or the illustration of the car and 
displayed. To display the passage locus of the body end 
having the appropriate height, the locus is connected to 
the car position corresponding to the height. 
[0095] Moreover, the car may be represented in the 
skeleton or the wire frame. In this case, the road surface 
passage locus 1903 may be drawn from the tire in the 
illustration. Thereby, the meaning of the road surface 
passage locus can easily be grasped by intuition. 
[0096] Furthermore, these auxiliary lines are dis- 
played together with the actual image or the illustration 
of the car, the road surface passage locus is displayed 
like the tire trace, or the bumper end passage locus is 
connected to the bumper end of the actual image or the 
illustration of the car and displayed. Thereby, the rela- 
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tion between two passage loci and the car may easily 
be grasped by intuition. 

[0097] Additionally, the mirror confirmation line is si- 
multaneously displayed, so that the driver can grasp the 
portion to be checked at the optimum timing by intuition. 

(Eleventh Embodiment) 

[0098] The constitution of the image synthesis/display 
apparatus according to an eleventh embodiment is the 
same as that of FIG. 1 8. FIG. 25 is a screen constitution 
diagram of the eleventh embodiment. The car will be de- 
scribed as a sedan hereinafter. 

[0099] in FIG. 25, an auxiliary line 2201 is a road sur- 
face passage locus indicating the position obtained by 
projecting the position passed by the body end of the 
car onto the road surface, when the car linearly runs 
backwards. An auxiliary line 2202 is a bumper end pas- 
sage locus indicating the position passed by the bumper 
end, when the car linearly runs backwards. An auxiliary 
line 2203 is a passage locus indicating the position 
passed by the body end of the car corresponding to the 
bumper position such as SUV having a bumper position 
higher than the bumper position of the car, when the car 
linearly runs backwards . Illustrations 2204A to 2204C 
are illustrations imitating car body surfaces of the rear 
part of the car, and states actually reflected in the image 
are shown as the illustrations when the car body rear 
part of the car is positioned in the respective positions. 
Additionally, the height of the illustration is assumed to 
be the same as the height of the auxiliary line 2203, but 
the height may not be necessarily the same, and the 
height of the auxiliary line 2203 may arbitrarily be deter- 
mined. 

[01 00] As described above, according to the eleventh 
embodiment of the present invention, the road surface 
passage locus of the car linearly running backwards, the 
bumper end passage locus of the car, the bumper end 
passage locus such as SUV, and the illustration imitating 
the car body of the rear part of the car are simultane- 
ously displayed. Therefore, the position relation be- 
tween the advancing direction of the car and the object 
present around the car can be grasped by intuition. 
[0101] Additionally, as described in the seventh and 
eighth embodiments, in combination with the illustration 
of the car, the illustration is moved or transformed for 
the screen, and used as the illustration imitating the car 
body surface of the rear part of the car, and the road 
surface passage locus is displayed like the tire trace. 
Thereby, the position relation between the advancing di- 
rection of the car and the object present around the car 
can be grasped by intuition. 

[0102] Moreover, the mirror confirmation line in the 
sixth embodiment may also be displayed. As in the tenth 
embodiment, the auxiliary lines are moved in accord- 
ance with the steering angle of the car, and it is also 
possible to display the locus passed during the actual 
backward running. Additionally, only the position of the 



auxiliary line is exactly displayed, and the illustration im- 
itating the rear part of the car may be selected from sev- 
eral predetermined types and superimposed upon the 
appropriate position. 

5 

(Twelfth Embodiment) 

[01 03] The constitution of the image synthesis/display ■• 
apparatus according to a twelfth embodiment is the 
10 same as that of FIG. 18. FIG. 26 is a screen constitution 
diagram of the twelfth embodiment. The car will be de-' 
scribed as a sedan hereinafter. 

[0104] In FIG. 26, an auxiliary line 2301 is a road sur- 
face passage locus indicating the position obtained by 

15 projecting the position passed by the body end onto the 
road surface, when the car linearly runs backwards. An 
auxiliary line 2302 is a bumper lower surface end pas- 
sage locus indicating the position passed by the end of 
the lower surface of the bumper, when the car linearly 

20 runs backwards. An auxiliary line 2303 is a bumper up- 
per surface passage locus indicating the position 
passed by the end of the upper surface of the bumper, 
when the car linearly runs backwards. Illustrations 
2304A to 2304C imitate the bumper, and a display state 

25 as the actual image is represented as the illustration, 
when only the bumper is present in the respective posi- 
tions. 

[0105] As described above, according to the twelfth 
embodiment of the present invention, the road surface . 

30 passage locus of the car linearly running backwards, the. _ 
. bumper upper surface end passage locus, the bumper. 
lower surface passage locus, and the illustration imitat- 
ing the bumper of the car are simultaneously displayed. 
Therefore, the position relation between the advancing 

35 direction of the car and the object present around the 
car can be grasped by intuition. 

[0106] Additionally, as in the seventh and eighth em- 
bodiments, in combination with the illustration of the car, 
- the illustration is moved or transformed for the screen, 

40 and used as the illustration imitating the car body sur- 
face of the rear part of the car, and the road surface pas- 
sage locus is displayed like the tire trace. Thereby, the 
position relation between the advancing direction of the 
car and the object present around the car can be 

45 grasped by intuition. 

[0107] Moreover, the mirror confirmation line in the 
sixth embodiment may also be displayed. As in the tenth 
embodiment, the auxiliary lines are moved in accord- 
ance with the steering angle of the car, and it is also 

50 possible to display the locus passed during the actual 
backward running. 

(Thirteenth Embodiment) 

55 [0108] FIG. 27 is a block diagram showing of the im- 
age synthesis/display apparatus according to a thir- 
teenth embodiment of the present invention. The em- 
bodiment includes the display device omitted in the 
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aforementioned embodiments. That is, as shown in FIG. 
27, there are provided: display data conversion control 
means 80 for receiving display data as an output signal 
of the car locus line drawing means 104 and converting 
the data to a signal having a predetermined form; and 5 
display means 81 for receiving the display data obtained 
by the display data conversion control means 80 and 
displaying the data on the screen. The display means 
81 may be a mirror having a display function, which is 
constituted by imparting the display function to a rear 10 
view mirror of the car. 

[0109] In the constitution, since the image is displayed 
in the mirror having the display function, the operator 
can easily recognize the surrounding situation and the 
car position. Additionally, the mirror having the display 15 
function is constituted by superposing a half mirror upon 
the surface of a display portion of the display means 81 
such as liquid crystal and organic electro luminescence 
(EL). The liquid crystal or organic EL portion may con- 
stitute one display portion, or an aggregate group of a 20 
plurality of display portions, and is constituted of at least 
one display portion. 

[0110] Furthermore, the mirror constitution can be 
changed over to a state for operating as a usual mirror, 
or a state for operating as an image display screen. In 25 
this changeover operation, a luminance sensor built in 
the car can automatically change the state to an image 
display state at night. Moreover, the operator may 
change the state in response to an operation for turning 
on a headlight. Moreover, the state may be changed to 30 
the image display state by turning on a thermal heater 
of a rear window, when the rear window fogs by rain, or 
the rear cannot be viewed by snow or icing in winter sea- 
son. Furthermore, for the changeover, in cooperation 
with a gear, the state may be changed to the image dis- 35 
play state only when the car is put in a reverse gear. 
[0111] As described above, the mirror whole surface 
can be changed to the mirror state or the image display 
state. Therefore, the operator can easily grasp the situ- 
ation behind the car with a wide display surface and can *o 
safely drive the car. 

[0112] As another mirror constitution replacing the 
aforementioned mirror constitution, the usual mirror is 
disposed together with the liquid crystal or organic EL 
display portion. In this replacing mirror constitution, the 45 
rear situation is roughly grasped with the usual mirror 
portion, and it is possible to enlarge and display a portion 
as a blind spot in the usual display portion, or to enlarge 
and display an object portion requiring particular atten- 
tion with the additionally disposed display portion. Addi- so 
tionally, these display means 81 are not limited to the 
mirror. Examples of the display means include a display 
portion disposed in a dashboard, a display connected 
to a navigation system, an information mobile terminal, 
a head-up display, and the like. 55 



(Fourteenth Embodiment) 

[01 13] FIG. 28 is a block diagram of an image acquire- 
ment/warning apparatus according to a fourteenth em- 
bodiment of the present invention. In the embodiment, 
in addition to the constitution of the thirteenth embodi- 
ment, there are provided: detection means 82 disposed 
in the rear part of the car; and warning means 83 re- 
sponding to an output signal of the detection means 82. 
The detection means 82 detects the abnormal approach 
of the objects such as a camping car being connected 
to the car as a connection object. When the distance 
between the car and the connection object is shorter 
than a predetermined distance, the abnormal approach 
can be judged. 

[01 14] By this constitution, an approach sensor as the 
detection means 82 is used to detect the approach of 
the camping car as the connection object, and an ap- 
proach state is confirmed with the image. When a dis- 
tance between a pair of load carrier hitches on a car side 
and on a camping car side is within a first predetermined 
distance such as one meter and a second predetermine 
distance such as 50 cm, a speaker as the warning 
means 83 can emit a warning sound to warn the oper- 
ator respectively. The warning sound can be varied. As 
described above, in the connection operation of the load 
carrier hitches, while the connection operation is 
checked with the image, an appropriate warning can be 
emitted in accordance with the approach state (in the 
distance arbitrarily set in accordance with the connec- 
tion operation), and the connection operation can safely 
and securely be performed. 

[0115] Additionally, the warning means 83 has been 
described as the speaker, but is not limited to the speak- 
er. The means is connected as shown by a dotted line 
of FIG. 28, and transmits a warning signal, so that a 
warning symbol image, a blinking image, or a color ap- 
propriate for calling an attention may be superimposed 
and displayed upon the image in the display device. Fur- 
thermore, alarming signal generation condition setting 
means 87 can be used to arbitrarily set a condition for 
issuing warning information in accordance with a size, 
weight, speed, and the like of the car or the connection 
object. 

[0116] In further constitution, the image pickup unit 
may be attached to the load carrier hitch, and a state of 
completion of the connection may be confirmed at the 
last moment. 

[01 17] Moreover, the detection means 82 may be ul- 
trasonic, electromagnetic, or optical (including infrared 
rays), or may be of a system in which the distance is 
directly calculated from the image and which issues a 
warning. 

[0118] Furthermore, the connection object may be 
other than a camping car like a load carrier with a board 
carried thereon, or the like. 
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(Fifteenth Embodiment) 

[0119] FIG. 29 is a block diagram of a car position rec- 
ognition apparatus according to a fifteenth embodiment. 
In the embodiment, in addition to the first to twelfth em- 5 
bodiments, the image, behind the car is processed, the 
image of the arbitrary object is detected, the position re- 
lation between the detected image and the image of the 
car is recognized, the recognized position relation is 
compared with the predetermined position relation, and 10 
a deviation amount between the relations is detected. 
That is, the car position recognition apparatus of the fif- 
teenth embodiment includes: image detection means 84 
for detecting the image of the arbitrary object from the 
image obtained by rear image pickup means of the im- is 
age pickup means 101 or the image synthesized by the 
picture synthesizing apparatus of any one of the first to 
twelfth embodiments; recognition means 85 for recog- 
nizing the position relation between the image detected 
by the image detection means 84 and the image of the 20 
car; and comparison means 86 for comparing the posi- 
tion relation recognized by the recognition means and 
the predetermined position relation and detecting the 
deviation amount from these position reliations. As the 
rear image pickup means, a camera disposed in the rear 25 
of the car may be used. Thereby, a white line on the road 
surface indicating a parking space behind the car is pho- 
tographed. The image detection means 84 detects the 
white line indicating the parking space from the image 
obtained by the rear image pickup means, the image 30 
projected onto a three-dimensional projection model ob- 
tained by combining a cylindrical surface and spherical 
surface, or the image synthesized by the image synthe- 
sizing means. The recognition means 85 calculates and 
recognizes the position relation between the detected 35 
white line and the image of the car from the image data. 
The comparison means 86 compares the actual position 
relation recognized by the recognition means 85 with the 
predetermined position relation in which a center of the 
image of the car is positioned in the middle between right *o 
and left white lines defining the parking space, and de- 
tects the deviation amount from these position relations. 
Additionally, when the deviation amount is abnormally 
large, the car is moved backwards, but the car cannot 
be parked between the right and left white lines. There- *5 
fore, the warning may be issued in the constitution. 
[0120] According to the constitution, the driver can 
confirm the white lines defining the parking space with 
the image while driving the car backwards, and can eas- 
ily grasp the position relation between the car and the 50 
parking space together with the position relation be- 
tween the car and the surrounding situation behind the 
car, so that safer driving is achieved. 
[0121] As described above, according to the present 
invention, the appropriate illustration, auxiliary line, to- 55 
cus line, and the like are superimposed upon the pseudb 
image from the virtual viewpoint. Thereby, the driver can 
grasp the three-dimensional position relation between 
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the car and the surrounding solid object by intuition. As 
a result, an appropriate contact judgment is facilitated. 
The picture synthesizing apparatus having these supe- 
rior effects can be provided. 

* • 

Claims 

1. A picture synthesizing apparatus comprising: im- 
age pickup means disposed in a car; viewpoint 
change image synthesizing means for changing a 
viewpoint of an image obtained by said image pick- 
up means and synthesizing the image; car locus 
line generation means for generating at least one 
of a locus line at an arbitrary height of said car and 
a vertical line; and car locus line drawing means for 
drawing the locus line generated by said car locus 
line generation means on the image synthesized by 
said viewpoint change image synthesizing means. 

2. The picture synthesizing apparatus according to 
claim 1 wherein said car locus line generation 
means comprises three-dimensional locus line gen- 
eration means, road surface projection means, and 
synthesized image projection means. 

3. The picture synthesizing apparatus according to 
claim 1 wherein said car locus line generation,, 
means generates a locus line in a case in which said . ! 
car linearly advances. 

— ■ 

4. The picture synthesizing apparatus according to 
claim 1, further comprising steering angle informa- 
tion output means for outputting a steering wheel 
angle of said car, wherein said car locus line gen- 
eration means generates the locus line in accord- 
ance with steering angle information outputted by 
said steering angle information output means. 

5. The picture synthesizing apparatus according to 
claim 1 which has a function of interpolating a locus 
line on a road surface of said car and the locus line 
at the arbitrary height with a straight line or a curved 
line, and drawing a line vertical to said road surface 
on said synthesized image. 

6. The picture synthesizing apparatus according to 
claim 1 which has a function of drawing a locus line 
of a bumper end of said car or a locus line of a car 
height on said synthesized image. 

7. The picture synthesizing apparatus according to 
claim 1 which has a function of changing a color or 
a thickness of said locus line in accordance with a 
distance from said car and drawing the locus line. 

8. The picture synthesizing apparatus according to 
claim 4 which has a function of drawing a section of 
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said car moved apart from a rear end of said car 
along said locus line with an elapse of time on said 
synthesized image. 

9. The picture synthesizing apparatus according to 
claim 4 which has a function of drawing a solid dia- 
gram of said car moved apart from a rear end of 
said car along said locus line with an elapse of time 
on said synthesized image. 

10. The picture synthesizing apparatus according to 
claim 4 wherein said car locus line generation 
means comprises three-dimensional Shape storage 

4 

means, three-dimensional locus region generation 
means, road surface projection means, and synthe- *5 
sized image projection means. 

11. The picture synthesizing apparatus according to 
claim 10 wherein said three-dimensional shape 
storage means stores a shape of said car. 20 

12. The picture synthesizing apparatus according to 
claim 10 wherein said three-dimensional shape 
storage means stores a shape of a rectangular par- 
allelepiped inscribed by said car. 25 

13. The picture synthesizing apparatus according to 
claim 10 wherein said three-dimensional shape 
storage means stores a shape of a wheel of said 
car. 30 

14. The picture synthesizing apparatus according to 
claim 10 wherein said three-dimensional shape 
storage means stores a shape of a bumper of said 
car. 35 

15. The picture synthesizing apparatus according to 
claim 4, further comprising obstacle collision predic- 
tion means for detecting an obstacle present 
around said car, and predicting a possibility of col- <*# 
lision of said car with said obstacle. 

16. The picture synthesizing apparatus according to 
claim 15 wherein said car locus line drawing means 
does not draw the locus line of said car ahead of a <*5 
collision place, when said obstacle collision predic- 
tion means predicts the collision of said car with said 
obstacle. 

17. The picture synthesizing apparatus according to so 
claim 15 wherein said car locus line drawing means 
emphasizes and displays a collision place, when 
said obstacle collision prediction means predicts 
the collision of said car with said obstacle. 

55 

18. The picture synthesizing apparatus according to 
claim 4, further comprising multi-screen generation 
means for displaying the image synthesized by said 
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viewpoint change image synthesizing means in a 
multiplicity of divided screens. 

19. The picture synthesizing apparatus according to 
5 claim 1 8 wherein said car locus line drawing means 

draws a locus of the car in each screen generated 
by said multi-screen generation means. 

20. The picture synthesizing apparatus according to 
10 claim 18 wherein said image pickup means includes 

means for picking up an image behind said car, and 
means for picking up an image beside said car. 

21. The picture synthesizing apparatus according to 
claim 18 wherein said car locus line drawing means 
draws a locus line of a rear end of said car on an 
image beside said car, or an image obtained by con- 
verting said image beside the car. 

22. The picture synthesizing apparatus according to 
claim 1 9 wherein said car locus line drawing means 
draws the locus line or a car frame indicating the 
same position in the same color in different screens, 
when said locus line is drawn in a plurality of 
screens. 

23. A picture synthesizing apparatus comprising: im- 
age pickup means disposed in a car; viewpoint 
change image synthesizing means for changing a 
viewpoint of an image obtained by said image pick- 
up means and synthesizing the image; auxiliary line 
generation means for generating an auxiliary line of 
an arbitrary position from said car; and auxiliary line 
drawing means for drawing the auxiliary line gener- 
ated by said auxiliary line generation means on the 
image synthesized by said viewpoint change image 
synthesizing means. 

24. The picture synthesizing apparatus according to 
claim 23, further comprising multi -screen genera- 
tion means for displaying the image synthesized by 
said viewpoint change image synthesizing means 
in a multiplicity of divided screens, wherein said 
auxiliary line drawing means draws the generated 
auxiliary line in each screen generated by said mul- 
ti-screen generation means. 

25. The picture synthesizing apparatus according to 
claim 23 wherein said auxiliary line generation 
means generates an auxiliary line indicating a po- 
sition of a rear end of said car. 

26. The picture synthesizing apparatus according to 
claim 23 wherein said auxiliary line generation 
means generates an auxiliary line indicating a con- 
stant distance from a rear end of said car. 

27. The picture synthesizing apparatus according to 
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claim 23 wherein said auxiliary line generation 
means generates an auxiliary line indicating a width 
of a general car. 

28. A picture synthesizing apparatus comprising: im- 5 
age pickup means disposed in a car so that a rear 
part of said car is positioned in a view field; view- 
point change image synthesizing means for chang- 
ing a viewpoint of an image obtained by said image 
pickup means and synthesizing the image including 10 
an image of said car; storage means for storing pre- 
determined data beforehand; and drawing means 

for superimposing predetermined auxiliary data up- 
on the image synthesized by said viewpoint change 
image synthesizing means based on the data read *5 
from said storage means. 

29. The picture synthesizing apparatus according to 
claim 28 which has a function of superimposing an 
auxiliary line upon a rear edge of said car, and pro- 20 
viding an image emphasizing/indicating the corre- 
sponding position. 

30. The picture synthesizing apparatus according to 
claim 28 which has a function of providing an image 25 
showing a three-dimensional illustration prepared 

as if the image of said car were picked up by an 
actually disposed image pickup unit and converted/ 
synthesized. 

30 

31. The picture synthesizing apparatus according to 
claim 28 which has a function of representing said 
car by an illustration of a skeleton or a wire frame, 
and providing an image explicitly indicating a tire 
position. 35 

32. The picture synthesizing apparatus according to 
claim 31 which has a function of providing an image 
obtained by transforming/synthesizing an image 
actually obtained by said image pickup means in a *o 
region corresponding to a bumper of the illustration. 

33. The picture synthesizing apparatus according to 
claim 28 which has a function of superimposing an 
illustration of two wall surfaces disposed vertically 45 
to a road surface in a rear end position of said car, 
and on an inner side of the rear end position of said 
car, and having the same width as a width of said 
car upon a displayed image, and providing an im- 
age metaphorically representing said car as a solid so 
object. 

34. The picture synthesizing apparatus according to 
claim 28 which has a function of providing an image 
showing a mirror confirmation line behind a rear end 55 
of a bumper of said car by a constant distance and 
horizontally with said bumper. 



35. The picture synthesizing apparatus according to 
claim 28 which has a function of providing an image 
including a road surface passage locus indicating a 
position obtained by projecting a position passed by 
a body end of said car onto a road surface, and a 
bumper end passage locus indicating a position 
passed by a bumper end of said car, when said car 
moves backwards, and a solid auxiliary line for con- 
necting the loci to produce a solid sense. 

36. The picture synthesizing apparatus according to 
claim 35 which has a function of providing an image 
showing said road surface passage locus like a tire 
trace, and indicating the bumper end passage locus 
connected to a bumper end of an actual image or 
an illustration of said car. 

37. The picture synthesizing apparatus according to 
claim 35 which has a function of providing an image 
showing said road surface passage locus drawn 
from a tire explicitly indicated in an illustration. 

38. The picture synthesizing apparatus according to 
claim 37 which has a function of providing an image 
showing a mirror confirmation line behind a rear end 
of a bumper of said car by a constant distance and 
horizontally with said bumper. 

39. The picture synthesizing apparatus according to 
claim 35 which further comprises a locus calculation 
unit to calculate a predicted locus from a steering 
angle signal inputted from the outside, and which 
has a function of providing an image including a 
road surface passage locus corresponding to a 
steering angle of said car, a bumper end passage 
locus corresponding to the steering angle of said 
car, and a solid auxiliary line for connecting the loci 
to produce a solid sense. 

40. The picture synthesizing apparatus according to 
claim 28 which has a function of providing an image 
simultaneously showing a road surface passage lo- 
cus, a bumper end passage locus, a passage locus 
indicating a position passed by an appropriate 
height portion of a body of said car, and an illustra- 
tion imitating a rear part of the car, when said car 
moves backwards. 

41. The picture synthesizing apparatus according to 
claim 40 which has a function of providing an image 
showing said road surface passage locus like a tire 
trace, and indicating said bumper end passage lo- 
cus connected to a bumper end of an actual image 
or an illustration. 

42. The picture synthesizing apparatus according to 
claim 40 which has a function of providing an image 
showing said road surface passage locus drawn 



15 



29 EP 1231 110 A2 



30 



from a tire explicitly indicated in an illustration. 

43. The picture synthesizing apparatus according to 
claim 42 which has a function of providing an image 
showing a mirror confirmation line behind a rear end 5 
of a bumper of said car by a constant distance and 
horizontally with said bumper. 

44. The picture synthesizing apparatus according to 
claim 40 which further comprises a locus calculation 10 
unit to calculate a predicted locus from a steering 
angle signal inputted from the outside, and which 
has a function of providing an image simultaneously 
showing said road surface passage locus corre- 
sponding to a steering angle of said car, said bump- is 
er end passage locus corresponding to the steering 
angle of said car, a passage locus indicating a po- 
sition passed by an appropriate height portion of a 
body of said car, and an illustration imitating a rear 
part of said car. 20 

45. The picture synthesizing apparatus according to 
claim 28 which has a function .of providing an image 
simultaneously showing a road surface passage lo- 
cus, a bumper upper surface passage locus indicat- 25 
ing a position passed by an end of a bumper upper 
surface of said car, a bumper lower surface pas- 
sage locus indicating a position passed by an end 

of a bumper lower surface of said car, and an illus- 
tration imitating a rear part of said car, when said 30 
car moves backwards. 

46. The picture synthesizing apparatus according to 
claim 45 which has a function of providing an image 
showing said road surface passage locus like a tire 35 
trace, and indicating the bumper end passage locus 
connected to a bumper end of an actual image or 

an illustration of the car. 

47. The picture synthesizing apparatus according to *o 
claim 45 which has a function of providing an image 
showing said road surface passage locus drawn 
from a tire explicitly indicated in an illustration. 

48. The picture synthesizing apparatus according to *5 
claim 47 which has a function of providing an image 
showing a mirror confirmation line behind a rear end 

of a bumper of said car by a constant distance and 
horizontally with said bumper. 

. so 

49. The picture synthesizing apparatus according to 
claim 45 which further comprises a locus calculation 
unit to calculate a predicted locus from a steering 
angle signal inputted from the outside, and which 
has a function of providing an image simultaneously 55 
showing said road surface passage locus corre- 
sponding to a steering angle of said car, said bump- 
er upper surface passage locus corresponding to 



the steering angle of said car, said bumper lower 
surface passage locus corresponding to the steer- 
ing angle of said car, and an illustration imitating a 
bumper of said car. 

50. An image synthesis/display apparatus comprising: 

a picture synthesizing apparatus comprising: 
image pickup means disposed in a car; view- 
point change image synthesizing means for 
changing a viewpoint of an image obtained by 
said image pickup means and synthesizing the 
image; car locus line generation means for gen- 
erating at least one of a locus line at an arbitrary 
height of said car and a vertical line; and car 
locus line drawing means for drawing the locus 
line generated by said car locus line generation 
means on the image synthesized by said view- 
point change image synthesizing means; 
display means for displaying the image synthe- 
sized by said picture synthesizing apparatus; 
and 

display data conversion means for converting 
said image to be displayed into data suitable 
for said display means. 

51. An image synthesis/display apparatus comprising: 

a picture synthesizing apparatus comprising: 
image pickup means disposed in a car; view- 
point change image synthesizing means for 
changing a viewpoint of an image obtained by 
said image pickup means and synthesizing the 
image; auxiliary line generation means for gen- 
erating an auxiliary line of an arbitrary position 
from said car; and auxiliary line drawing means 
for drawing the auxiliary line generated by said 
auxiliary line generation means on the image 
synthesized by said viewpoint change image 
synthesizing means; 

display means for displaying the image synthe- 
sized by said picture synthesizing apparatus; 
and 

display data conversion means for converting 
said image to be displayed into data suitable 
for said display means. 

52. An image synthesis/display apparatus comprising: 

a picture synthesizing apparatus comprising: 
image pickup means disposed in a car so that 
a rear part of said car is positioned in a view 
field; viewpoint change image synthesizing 
means for changing a viewpoint of an image ob- 
tained by said image pickup means and syn- 
thesizing the image including an image of said 
can storage means for storing predetermined 
data beforehand; and drawing means for su- 
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perimposing predetermined auxiliary data upon 
the image synthesized by said viewpoint 
change image synthesizing means based on 
the data read from said storage means; 
display means for displaying the image synthe- 5 
sized by said picture synthesizing apparatus; 
and 

display data conversion means for converting 
said image to be displayed into data suitable 
for said display means. 10 

53. An image acquirement warning apparatus compris- 
ing: 

detection means for detecting an approaching 15 
state of a connection object connected to a car; 
a picture synthesizing apparatus comprising: 
image pickup means disposed in said car and/ 
or said connection object; viewpoint change im- 
age synthesizing means for changing a view- 20 
point of an image obtained by said image pick- 
up means and synthesizing the image; car lo- 
cus line generation means for generating at 
least one of a locus line at an arbitrary height 
of said car and a vertical line; and car locus line 25 
drawing means for drawing the locus line gen- 
erated by said car locus line generation means 
on the image synthesized by said viewpoint 
change image synthesizing means; and 
warning means for generating a warning signal 30 
from said approaching state obtained by said 
detection means and/or a relation between said 
car and said connection object in the image 
synthesized by said picture synthesizing appa- 
ratus. 35 

54. An image acquirement warning apparatus compris- 

* 

mg: 

detection means for detecting an approaching 40 
state of a connection object connected to a car; 
a picture synthesizing apparatus comprising: 
image pickup means disposed in said car and/ 
or said connection object; viewpoint change imr 
age synthesizing means for changing a view- 45 
point of an image obtained by said image pick- 
up means and synthesizing the image; auxiliary 
line generation means for generating an auxil- 
iary line of an arbitrary position from said car; 
and auxiliary line drawing means for drawing 50 
the auxiliary tine generated by said auxiliary 
line generation means on the image synthe- 
sized by said viewpoint change image synthe- 
sizing means; and 

warning means for generating a warning signal 55 
from said approaching state obtained by said 
detection means and/or a relation between said 
car and said connection object in the image 



synthesized by said picture synthesizing appa- 
ratus. 

55. An image acquirement warning apparatus compris- 
ing: 

detection means for detecting an approaching 
state of a connection object connected to a car; 
a picture synthesizing apparatus comprising: 
image pickup means disposed in said car and/ 
or said connection object so that a rear part of 
said car is positioned in a view field; viewpoint 
change image synthesizing means for chang- 
ing a viewpoint of an image obtained by said 
image pickup means and synthesizing the im- 
age including an image of said car; storage 
means for storing predetermined data before- 
hand; and drawing means for superimposing 
predetermined auxiliary data upon the image 
synthesized by said viewpoint change image 
synthesizing means based on the data read 
from said storage means; and 
warning means for generating a warning signal 
from said approaching state obtained by said 
detection means and/or a relation between said 
car and said connection object in the image 
synthesized by said picture synthesizing appa- 
ratus. 

56. The image acquirement warning apparatus accord- . 
ing to any one of claims 53 to 55, further comprising 
warning signal generation condition setting means 
for a user to arbitrarily set a condition for generating 
the warning signal by said warning means. 

57. A car position recognition apparatus comprising: 

a picture synthesizing apparatus comprising: a 
plurality of image pickup means disposed in a 
car, and including rear image pickup means for 
picking up an image behind said car; viewpoint 
change image synthesizing means for chang- 
ing a viewpoint of an image obtained by said 
image pickup means and synthesizing the im- 
age; car locus line generation means for gen- 
erating at least one of a locus line at an arbitrary 
height of said car and a vertical line; and car 
locus line drawing means for drawing the locus 
line generated by said car locus line generation 
means on the image synthesized by said view- 
point change image synthesizing means; 
image detection means for detecting an image 
of an arbitrary object from the image obtained 
by said rear image pickup means or the image 
synthesized by said picture synthesizing appa- 
ratus; 

recognition means for recognizing a position re- 
lation between the image detected by said im- 
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age detection means and the image of said car; 
and 

comparison means for comparing said position 
relation recognized by said recognition means 
with a predetermined position relation, and de- 5 
tecting a deviation amount between the posi- 
tion relations from the position relations. 

58. A car position recognition apparatus comprising: 

10 

a picture synthesizing apparatus comprising: a 
plurality of image pickup means disposed in a 
car, and including rear image pickup means for 
picking up an image behind said car; viewpoint 
change image synthesizing means for chang- 15 
ing a viewpoint of an image obtained by said 
image pickup means and synthesizing the im- 
age; auxiliary line generation means for gener- 
ating an auxiliary line of an arbitrary position 
from said car; and auxiliary line drawing means 20 
for drawing the auxiliary line generated by said 
auxiliary line generation means on the image 
synthesized by said viewpoint change image 
synthesizing means; 

image detection means for detecting an image 25 
of an arbitrary object from the image obtained 
by said rear image pickup means or the image 
synthesized by said picture synthesizing appa- 
ratus; 

recognition means for recognizing a position re- 30 
lation between the image detected by said im- 
age detection means and the image of said car; 
and 

comparison means for comparing said position 
relation recognized by said recognition means 35 
with a predetermined position relation, and de- 
tecting a deviation amount between the posi- 
tion relations from the position relations. 

59. A car position recognition apparatus comprising: 40 
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by said rear image pickup means or the image 
synthesized by said picture synthesizing appa- 
ratus; 

recognition means for recognizing a position re- 
lation between the image detected by said im- 
age detection means and the image of said car; 
and 

comparison means for comparing said position 
relation recognized by said recognition means 
with a predetermined position relation, and de- 
tecting a deviation amount between the posi- 
tion relations from the position relations. 



a picture synthesizing apparatus comprising: a 
plurality of image pickup means disposed in a 
car, including rear image pickup means for pick- 
ing up an image behind said car, and disposed 45 
so that a rear part of said car is positioned in a 
view field; viewpoint change image synthesiz- 
ing means for changing a viewpoint of an image 
obtained by said image pickup means and syn- 
thesizing the image including an image of said 50 
car; storage means for storing predetermined 
data beforehand; and drawing means for su- 
perimposing predetermined auxiliary data upon 
the image synthesized by said viewpoint 
change image synthesizing means based on 55 
the data read from said storage means; 
image detection means for detecting an image 
of an arbitrary object from the image obtained 
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FIG. 4 
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